Frame-rate up-conversion using transmitted true motion vectors

In this paper, we present a video frame-rate up-conversion scheme that uses transmitted true motion vectors for motion-compensated interpolation. In past work, we demonstrate that a neighborhood-relaxation motion tracker can provide more accurate true motion information than a conventional minimal-residue block-matching algorithm. Although the technique to estimate the true motion vectors is a novelty in its own right, the strength of this technique can be further demonstrated through various spatio-temporal interpolation applications. In this work, we focus on the particular problem of frame-rate up-conversion. In the proposed scheme, the true motion field is derived by the encoder and transmitted by normal means (e.g., MPEG or H.263 encoding). Then, it is recovered by the decoder and is used not only for motion compensated predictions but also used to reconstruct missing data. It is shown that the use of our neighborhood-relaxation motion estimation provides a method of constructing high quality image sequences in a practical manner. INTRODUCTION Low bit-rate video compression plays an significant role in many multimedia applications, such as video-conferencing, video-phone, and video games. The proposed frame-rate up-conversion scheme provides better picture quality in playing back highly-compressed video. To accomplish acceptable coding results at low bit-rates, most encoders reduce the temporal resolution. In other words, instead of targeting the full frame rate of 30 frames/sec (fps), the frame rate may be reduced to 10 fps, which would mean that 2 out of every 3 frames are never even considered by the encoder. However, to display the full frame rate at the decoder, a recovery mechanism is needed. The simplest recovery mechanism would repeat each frame until a new frame is received. The problem is that the image sequence will appear jerky, especially in areas of large or complex motion. Another simple mechanism is to linearly interpolate between coded frames. The problem with this mechanism is that the image sequence will appear blurry in areas of motion. This type of artifact is commonly referred to as a ghost artifact. From the above, it is motion that the major cause of problems for image recovery of this kind. This has been observed by a number of researchers [1, 4, 5, 6], and it has been shown that a motion-compensated interpolation can provide better results. In [6], up-sampling results are presented using decoded frames at low bitrates. However, the receiver needs to perform a separate motion estimation just for the interpolation. In [5], the proposed algorithm considers multiple motion vectors for a single block so as to provide better picture qualities. However, this scheme In IEEE Workshop on Multimedia Signal Processing, Los Angeles, CA, Dec. 1998.